docs: add SDS protocol for scalable e2e reliability #108
Conversation
| ```protobuf | ||
| syntax = "proto3"; | ||
|
|
||
| message Message { |
There was a problem hiding this comment.
It may actually make sense to have a LogEntry or Entry terminology in this spec because the term message is very generic, and actually already a bit confusing in the status-go codebase.
Or maybe doing like WakuMessage and prefix/postfix to help with setting the context.
There was a problem hiding this comment.
This is a good point, although I'm struggling to find a term that wouldn't create confusion elsewhere. "Log" and "log entries" are already overloaded terms too and the mental model of "messages = log entries" may not be immediately apparent to someone approaching the code. Perhaps just calling it a SyncedMessage? Wdyt?
|
|
||
| message Message { | ||
| //1 Reserved for sender/participant id | ||
| string message_id = 2; // Unique identifier of the message |
There was a problem hiding this comment.
Concretely, is that a Waku message id or a Status envelop id?
I assume Waku message id because one needs to be able to retrieve it from a store node?
But then if you cannot find the message from a Waku store, you can't have a re-transmission of it because it would change the hash of the message (timestamp is included).
There was a problem hiding this comment.
This would be a Status-level ID as the Waku message ID would (well, should) be inaccessible at this layer. A mapping could be done from SDS message ID to Waku message hash that gets updated with each retransmission.
| If dependencies are unmet, | ||
| the participant MUST add the message to the incoming buffer of messages with unmet causal dependencies. | ||
|
|
||
| #### Deliver Message |
There was a problem hiding this comment.
Could it be Record message instead? I understand this step is about inscribing the message to the local log to handover to the application. or Transcribe.
There was a problem hiding this comment.
In this case we chose "deliver" as it matches terminology in related literature (specifically here: https://arxiv.org/pdf/2012.00472)
vac/raw/sds.md
Outdated
|
|
||
| #### Periodic Incoming Buffer Sweep | ||
|
|
||
| The participant MUST periodically check causal dependencies for each message in the incoming buffer.- For each message in the incoming buffer: |
There was a problem hiding this comment.
| The participant MUST periodically check causal dependencies for each message in the incoming buffer.- For each message in the incoming buffer: | |
| The participant MUST periodically check causal dependencies for each message in the incoming buffer. For each message in the incoming buffer: |
|
|
||
| #### Periodic Outgoing Buffer Sweep | ||
|
|
||
| The participant MUST rebroadcast **unacknowledged** outgoing messages after a set period. |
There was a problem hiding this comment.
Waku message hash changes when re-broadcast. How is this handled?
There was a problem hiding this comment.
Out of scope of this spec, as it focuses on the SDS protocol in isolation.
In practice, the various Waku "transport level" hashes should be mapped to the SDS-level message ID. When a message gets rebroadcast, the Waku hash may be updated for the same message ID entry.
There was a problem hiding this comment.
SDS-level message ID what is it in practical term in Status context?
Co-authored-by: fryorcraken <[email protected]>
Co-authored-by: Jimmy Debe <[email protected]>
Introduces the "Scalable Data Sync" protocol, based on the e2e reliability mechanisms discussed in the Vac forum post.
The initial aim is to provide an end-to-end reliability protocol for Status chat protocols, specifically within the context of Status Communities.